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ABSTRACT 


This  paper  explores  analysis  in  intelligence  and  evaluates  a  proposal  to  certify  and  convey 
analytical  rigor  as  it  relates  to  intelligence  products.  To  accomplish  this,  an  examination  of 
intelligence  failures  is  conducted  to  assess  the  application  of  analytical  rigor  across  historic  case 
studies.  The  historic  failures  demonstrate  gaps  in  standardization  and  insufficiencies  in 
analytical  rigor.  This  paper  proposes  establishing  unit  level  certified  Masters  of  Analytic 
Tradecraft  (MAT)  analysts  to  be  trained  and  entrusted  to  evaluate  and  rate  the  standards  and 
analytical  rigor  of  intelligence  products  prior  to  publication.  This  certification  model  is  based  on 
research  of  the  structure  of  professional  civilian  corporations.  Implementing  the  MAT  analyst 
certification  will  allow  decision-makers  the  ability  to  judge  when  analysis  is  robust  or  shallow  so 
that  they  can  meter  their  trust  accordingly.  This  certification  will  hold  the  IC  accountable  in  their 
adherence  to  the  established  analytical  standards,  ensure  compliance  is  properly  conveyed,  and 
promote  a  quality  of  analysis. 
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Section  1:  INTRODUCTION 


Research  has  shown  the  difficulty  in  recognizing  inadequacies  of  analytical  rigor  when 
consuming  intelligence  analysis  within  information  domains.1  For  the  purposes  of  this  paper, 
analytical  rigor  is  defined  as  “an  emergent  multi-attribute  measure  of  sufficiency.”2  The  term 
analytic  rigor  is  used  to  convey  the  level  of  analytic  tradecraft  application:  sometimes  robust 
and  other  times  weak.  Decision  makers  should  be  provided  “with  actionable  data  backed  by 
analytical  rigor.”3  When  evaluating  a  problem-set,  an  intelligence  consumer’s  perceived 
analytical  rigor  (based  on  perceived  cues)  ideally  should  meet  or  exceed  effective  rigor  (based 
on  analytical  process).4  To  accomplish  this,  decision  makers  should  not  be  left  to  their  own 
perceptions  in  judging  the  analytic  rigor  of  given  intelligence  product,  they  need  to  be  provided 
fidelity  on  the  level  of  analytical  rigor;  they  need  to  know  when  analysis  is  robust  or  shallow  so 
that  they  can  meter  their  trust  accordingly. 

Unfortunately,  examples  of  high-consequence,  high  profile  intelligence  fails  have  been 
common,  revealing  a  consistent  lack  of  analytical  sufficiency  and  raising  concerns  about  the 
condition  of  United  States  (U.S.)  intelligence.5  The  2003  assessment  of  Iraq’s  weapons  of 
mass  destruction  (WMD)  is  one  example  of  a  catastrophic  failure  on  the  part  of  the  IC.6  The 
WMD  Commission  Report  summarized  the  assessment  failures  that  informed  the  2003  invasion 
of  Iraq  in  this  way:  “This  failure  was  in  large  part  the  result  of  analytical  shortcomings; 
intelligence  analysts  were  too  wedded  to  their  assumptions  about  Saddam’s  intentions...  We 
conclude  that  the  Intelligence  Community  was  dead  wrong  in  almost  all  of  its  pre-war 
judgments  about  Iraq’s  weapons  of  mass  destruction...  [there  existed]  serious  errors  in 
analyzing. . .  [and  a]  failure  to  make  clear  just  how  much  of  its  analysis  was  based  on 
assumptions,  rather  than  good  evidence. .  .”7 
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Where  it  is  typical  for  analysts  to  use  tradecraft  language  to  convey  likeliness  and 
confidence,  no  such  effort  is  afforded  to  articulating  analytical  rigor.  If  an  assurance  of 
standards  and  a  conveyed  level  of  rigor  accompanied  assessment,  it  would  empower  decision 
makers  as  they  evaluate  and  measure  the  level  of  confidence  they  should  apply  to  the 
assessment.  U.S.  law  has  mandated  that  that  the  Office  of  the  Director  of  National  Intelligence 
(ODNI)  conduct  post-production,  after-action  quality  assurance  spot  checks,  however  there  is 
not  a  universal  method  or  qualifying  standard  of  analytical  rigor  prior  to  the  publication  of 
intelligence  reports.  Perceived  levels  of  rigor  can  be  misled  by  the  appearance  product  quality, 
with  the  perception  of  “sufficient  rigor”  evolving  once  insight  to  the  analytical  process  is 
provided. 8 

The  application  of  a  certification  model  will  challenge  analysts  and  organizations  to  meet 
higher  levels  of  analytical  rigor  while  providing  process  insight  to  decision  makers.  In  the 
civilian  world  certifications  are  used  to  help  regulate  professional  fields  such  as  medicine,  law, 
and  engineering  by:  1)  establishing  minimally  acceptable  standards,  2)  providing  accountability 
and  reassuring  consumers  that  the  professionals  are  deserving  of  trust,  3)  improving  quality  by 
providing  guidance  and  by  capturing  and  disseminating  best  practices.9  The  U.S.  Green 
Building  Council  (USGBC),  an  organization  that  promotes  sustainability-focused  construction 
practices,  has  a  unique  approach  in  their  Leadership  in  Energy  and  Environmental  Design 
(LEED)  program.10  USGBC  not  only  credentials  its  associates  (builders,  architects,  designers), 
but  has  a  four  tiered  rating  system  to  certify  standards  of  construction  for  a  given  project.  The 
LEED  tiered  rating  system  quantifies  and  clearly  conveys  that  standard  to  consumers. 

This  paper  will  evaluate  the  achievability  of  certifying  intelligence  products  by  first 


2 


providing  context  and  initial  historical  and  current  analysis  of  intelligence  practices.  It  will 
define  sufficiency  of  analytical  rigor  by  evaluating  its  independent  attributes  and  analyzing  how 
ODNI,  and  the  Air  Force  guidance  define  analytical  standards.  A  proposal  for  both  an  analyst 
and  pre-publication  product  certification  process  will  be  presented  and  evaluated  to  assess  if  a 
certification  process  should  be  incorporated  across  the  IC  and  whether  or  not  that  process  would 
or  would  not  affect  how  analytical  rigor  is  conveyed  to  decision  makers. 

Section  2:  BACKGROUND 

Today,  seventeen  agencies  and  organizations  comprise  the  IC,  with  over  1,200  sub¬ 
organizations  and  over  1,900  private  companies,  comprising  a  50+  billion  dollar  budget  and 
employ  over  800,000  people  with  TOP  SECRET  clearances.11  Its  collective  analytical  might 
culminates  in  intelligence  reports  like  the  President’s  Daily  Brief  ( PDB),  Worldwide 
Intelligence  Review  (WIRe),  National  Intelligence  Estimate  (NIE),  Secretary  of 
Defense/Chairman  of  the  Joint  Chiefs  of  Staff  Daily  Intelligence  Update  (SECDEF/CJCS  J2 
Daily  Intelligence  Update),  and  the  Secretary’s  Morning  Summary  (SMS).12  The  IC’s 
effectiveness  has  been  the  topic  of  much  investigation.  How  analysis  is  accomplished  and 
communicated  varies  greatly  across  the  IC’s  17  agencies  and  organizations,  which  continually 
battle  tendencies  to  stovepipe  information  behind  shrouds  of  secrecy  and  “need-to-know” 
compartmentalized  barriers.  Many  experts  have  pointed  to  an  IC  that  is  obsessed  with  the 
churn  of  data  and  overly  focused  on  the  tactical  and  current  intelligence  needed  to  impact  the 
tactical  battlefield  in  real-time;  going  as  far  to  reference  this  phenomena  as  “tyranny”:  the 
tyranny  of  current  intelligence,  tyranny  of  taskings,  tyranny  of  daily  operations,  and  the  tyranny 
of  the  immediate  to  name  a  few. 13  Although  valued  by  its  consumers,  a  focus  on  the  immediate 
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has  detracted  from  the  strategic  intelligence  priorities  that  inform  national  level  decision  makers 
and  requires  deep  understanding  required  for  strategic,  long-term  analysis.14  The  propagation 
of  information  and  the  availability  of  real-time  interconnected  data  has  “exacerbated  the  risk  of 
shallow  information  analysis.”15  Where  a  tactical  real-time  focus  can  save  lives  and  assists  in 
capturing  adversaries  on  the  battlefield,  strategic  intelligence  has  the  power  to  both  justify  and 
deter  wars.  Under  production  pressures,  this  collection-driven,  real-time  focus  is  much  desired 
by  war-fighters,  albeit  at  the  detriment  of  deep  understanding;  lacking  the  sustained  expert 
scrutiny  required  to  do  the  long-term  analysis  necessary  to  evaluate  and  inform  strategic 
problem-sets.  Tactical  and  strategic  assessments  are  not  mutually  exclusive,  for  they  inform 
one  another.  A  strategic,  deep  understanding  of  a  target-system  or  a  target-country  is  informed 
through  the  aggregation  of  hundreds,  if  not  thousands  of  tactical  and  operational  assessments. 

The  2008  RAND  repot,  Assessing  the  Tradecraft  of  Intelligence  Analysis,  phrases  this 
tension  as  “intelligence  reporting”  verses  “intelligence  analysis;”  intelligence  reporting 
requiring  little  if  any  critical  thinking,  application  of  tradecraft,  or  utilization  of  structured 
analytical  techniques.16  This  same  report  suggested:  “The  need  for  a  focal  point  in  analysis, 
and  analytic  tradecraft  is  striking,  and  this  need  will  only  grow  as  the  Community  strives  to  be 
more  ‘joint’  in  the  wake  of  the  December  2004  intelligence  reform  law  and  the  creation  of  a 
director  of  national  intelligence.”17  Community-wide  standards  are  essential  for  analysts  to  be 
effective  in  how  they  communicate  assessments  to  their  customers. 

The  “intelligence  process”  pulls  collection  from  the  operational  environment,  takes  that 
data  and  processes  it  into  information. 18  Once  put  through  the  rigors  of  analysis,  information  then 
becomes  intelligence.  Actionable,  timely,  and  relevant  intelligence  informs  national  decision 
makers  and  the  warfighter  consumers.  This  requirement  driven  intelligence  process  is  enabled 
through  analysis,  which  is  the  “thinking”  part  of  intelligence. 19  Analysis  is  the  pursuit  of 
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situational  understanding  and  thus  facilitates  the  goal  of  bold,  predictive,  and  informative 
assessments.  The  tradecraft  of  analysis  is  the  underpinning  of  the  intelligence  cycle  and  is  key  to 
the  development  of  deliverable  intelligence.20 

A  review  of  intelligence  failures  has  shown  deficiencies  in  analysis  to  be  a  consistent  cause 
of  IC  shortcomings.21  Poor  application  of  analytical  tradecraft  standards,  not  employing  structured 
analytic  techniques,  and  applying  levels  of  mediocre  analytical  rigor  have  caused  strategic 
assumptions,  dating  back  over  a  half  century,  which  were  not  appropriately  challenged.  Some 
examples  include:  the  1962  Cuban  Missile  Crisis,  the  1973  Yom  Kippur  War,  the  1989  German 
Unification,  and  the  1998  Indian  Nuclear  Test.22 

Effective  analysis  is  the  key  to  preventing  shortfalls  such  as  those  highlighted  in  the 
surprise  attacks  of  September  11th.  The  9-11  Commission  relied  on  Staff  Statement  No.  11,  The 
Performance  of  the  Intelligence  Community  to  inform  their  understanding  of  the  IC’s  performance 
leading  up  to  the  September  1 1  attacks  on  the  United  States.  This  document  described  how  the 
IC’s  emphasis  on  analysis  waned  in  the  1990’s  as  budgets  cuts  prevailed  and  competition  with 
operations  and  collection  further  detracted  from  the  importance  of  analysis.23  Analysis  competed 
with  the  introduction  of  the  24-hour  news  cycle  and  analytical  production  was  driven  away  from 
strategic,  long-range,  deep  analysis  and  instead  shifted  towards  current,  tactical  intelligence.  The 
Intelligence  Reform  and  Terrorism  Prevention  Act  of  2004  (IRTPA)  acted  on  these  shortfalls  and 
mandated  that  the  Director  of  National  Intelligence  convey  analytical  tradecraft  standards  and 
implement  processes  to  ensure  that  the  Intelligence  Community  production  meets  a  standard  of 
high  analytical  rigor.24 

An  analysis  of  the  2002  National  Intelligence  Estimate  (NIE)  on  Iraq’s  weapons  of  mass 
destruction  (WMD)  and  the  prewar  intelligence  leading  up  to  the  invasion  of  Iraq  was 
accomplished  by  the  Senate  Select  Committee  on  Intelligence  (SSCI),  which  exposed  significant 
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analytical  shortcomings. 25  This  extensive  review  highlighted  that  groupthink  dynamics,  failure  to 
challenge  key  assumptions,  and  instances  of  low  rigor  analysis  that  built  on  previous  reports 
without  critically  challenging  their  findings  were  all  contributing  factors  that  led  to  a  misleading, 
low-rigor  analytical  assessment.26  An  additional  review,  the  2005  Commission  on  the  Intelligence 
Capabilities  of  the  United  States  Regarding  Weapons  of  Mass  Destruction  (The  Robb-Silberman 
Report),  confirmed  the  need  to  more  thoroughly  train  analysts  on  analytical  tradecraft  requirements 
and  emphasized  the  need  for  analysts  to  challenge  assumptions  through  heightened  analytical 

27 

rigor.  7 

Recent  intelligence  case  studies  further  highlight  the  need  for  increased  analytical  rigor. 
These  cases  include  how  the  IC  failed  to  predict  the  2012  attacks  on  the  U.S.  Embassy  in  Benghazi, 
underestimated  the  2014  rise  of  the  Islamic  State  in  Iraq,  and  failed  to  predict  the  2014  Russian 
invasion  of  Crimea.  The  retrospective  review  of  these  recent  examples  is  still  ongoing  and  has  yet 
to  be  fully  evaluated.  The  exception  is  the  2014  Senate  Intelligence  Committee  report,  S.  Rpt.  113- 
134  Review  of  the  Terrorist  Attacks  on  U.S.  Facilities  in  Benghazi,  Libya,  September  11-12,  2012, 
which  is  one  of  seven  investigations  on  the  topic  containing  varying  degrees  of  focus  on  the  State 
Department  and  specifically  Secretary  Hillary  Clinton’s  role.  This  particular  report  focuses  on  the 
analysis  and  actions  of  the  IC  and  poor  application  of  rigor  and  lack  of  emphasis  in  conducting 
open-source  analysis.28 

The  lessons  from  these  failures  require  evaluation  and  application  to  the  ever-changing  and 
challenging  intelligence  environment  of  today.  Analysts  should  not  only  understand  their  target 
systems  and  target  countries  through  their  available  collection  sources;  they  must  also  be  able  to 
identify  nuanced  inferences  and  cultural  undertones  in  their  analysis.  Operating  in  an  environment 
of  fragmented  ambiguity,  with  undefined  adversaries,  they  are  presented  with  challenging  and 
evolving  problem- sets.  Analysts  are  expected  to  provide  thorough  analysis  with  limited  time, 
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scarce  resources,  evolving  technologies,  complex  geostrategic  conditions  and  extremely  agile 
adversaries.  The  IC  is  asked  to  be  predictive,  assessing  intent  and  understanding  the  probability  of 
events  that  have  yet  to  occur.  The  IC  is  operating  under  the  pressures  of  ongoing,  time  critical 
wars,  and  is  continually  fighting  to  identify  and  prevent  the  next  catastrophic  event.  The  IC  strives 
to  be  right  100%  of  the  time,  where  the  adversary  only  needs  to  be  right  once.  It  is  a  daunting 
challenge,  which  requires  adaptability  and  agility.  It  is  a  challenge  that  must  be  approached 
methodically  with  a  sustained  and  communicated  application  of  analytical  rigor. 

The  Intelligence  Reform  and  Terrorism  Prevention  Act  of  2004  (IRTPA)  provided  a 
foundational  change  in  its  mandate  to  establish  a  Director  of  National  Intelligence  (DNI)  “to  be 
responsible  for  ensuring  that  finished  intelligence  products  produced  by  any  element  or  elements  of 
the  intelligence  community  are  timely,  objective,  independent  of  political  considerations,  based 
upon  all  sources  of  available  intelligence,  and  employ  the  standards  of  proper  analytic  tradecraft”.29 
IRTPA  additionally  mandated  for  the  DNI  to  assign  an  individual  responsible  to  establish  processes 
to  ensure  “alternative  analysis”  (commonly  referred  to  as  structured  analytic  techniques  (SATs)), 
and  perform  regular  reviews  of  finished  intelligence  products  in  order  to  “draft  lessons  learned, 
identify  best  practices,  or  make  recommendations  for  improvement  to  the  analytic  tradecraft.”  The 
Office  of  the  Director  on  National  Intelligence  established  a  Deputy  Director  of  National 
Intelligence  for  Intelligence  Integration  (DDNI/II)  who  then  established  the  Mission  Integration 
Division’s  Analytic  Integrity  and  Standards  group  (AIS).  The  AIS  was  established  to  stimulate 
analytic  integrity,  promote  rigor,  encourage  learning,  and  institutionalize  tradecraft  across  the 
analytic  community.30  The  ODNI’s  Analytic  Integrity  and  Standards  group  has  a  unique 
requirement  to  evaluate  samples  of  the  IC’s  analytical  products  for  adherence  to  the  nine  analytic 
tradecraft  standards  identified  in  ICD  203.  To  do  this,  AIS  staff  evaluators  are  thoroughly  trained 
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(but  not  certified)  to  use  a  published  rating  scale  to  ensure  that  they  evaluate  sampled  intelligence 
products  and  analysis  in  a  fair,  consistent,  measured,  and  largely  unbiased  manner. 

On  21  June  2007,  the  AIS  presented  a  milestone  effort  in  the  publication  of  Intelligence 
Community  Directive  203,  Analytic  Standards  (ICD  203).  ICD  203  was  intended  to  serve  as  a 
foundational  document,  seeding  IC  training  and  education  initiatives  while  promoting  a  culture  of 
analytic  rigor  and  excellence.  Additionally,  ICD  203  serves  as  the  pre-production  baseline,  and  the 
post-intelligence  standard  by  which  the  ODNI  evaluates  IC  analytic  products.  While  the  ICD  203 
presents  14  different  analytic  and  analytic  tradecraft  standards,  it  is  a  mere  five  and  a  half  pages 
long,  including  introductory  materials  and  a  section  outlining  the  DDNI//II  responsibilities.  ICD 
203  provides  the  expectations  of  analytic  standards,  but  does  little  to  describe  how  these  standards 
should  be  applied.  As  a  compliment  to  ICD  203,  the  ODNI  has  published  a  Rating  Scale  for 
Evaluating  Analytic  Tradecraft  Standards,  which  is  primarily  intended  to  guide  ODNI  evaluators 
in  their  analysis  of  post-production  intelligence  products,  but  also  encouraged  as  a  guide  for 
analysts  to  enhance  their  tradecraft.  As  an  impromptu  survey  to  judge  penetration  of  these  ODNI 
initiatives,  thirty  professional  Air  Force  intelligence  analysts,  officers  and  enlisted,  across  three 
unique  and  independent  squadrons  (two  geospatial  targeting  squadrons,  two  cyber  intelligence, 
reconnaissance,  and  surveillance  squadrons,  and  one  remotely  piloted  aircraft  squadron)  were 
asked  if  they  knew  of  the  existence  of  these  foundational,  standard  setting  documents;  none  had 
any  awareness  of  ICD  203  or  the  rating  scale,  evaluation  document. 

With  the  29  March  2016  publication  of  Air  Force  Instruction  14-133,  Intelligence  Analysis 
(AFI  14-133),  the  Air  Force  is  set  to  improve  organization-wide  visibility  of  analytical  standards. 
AFI 14-133  is  the  Air  Force’s  attempt  to  recognize  the  foundational  importance  of  analysis  and  to 
codify  standards,  roles,  and  responsibilities.  It  focuses  primarily  on  Air  Force  level,  Major 
Command  (MAJCOM)  and  National  Air  and  Space  Intelligence  Center  roles  and  responsibilities. 
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How  this  regulation  will  translate  to  unit  level  analysis,  adaption  rates,  and  cultural  change  are  all 
unknowns  as  it  relates  to  the  Air  Force’s  effort  to  translate  ICD  203 ’s  tradecraft  standards  across 
the  entirety  AF ISR  Enterprise. 

While  Congressional  reports  have  identified  issues  at  a  surface  level,  an  abundance  of 
additional  research  conducted  in  the  timeframe  since  9-1 1  has  provided  both  retrospective  and 
forward  thinking  assessments  on  the  state  of  analysis  across  the  IC.  The  9-11  Commission  Report 
states  that  there  is  a,  “Lack  of  common  standards  and  practices  across  the  foreign-domestic 
divide. .  .holding  the  work — wherever  it  is  done — to  a  common  standard  of  quality  in  how  it  is 
collected,  processed  (e.g.,  translated),  reported,  shared,  and  analyzed...”31  The  2008  RAND  report 
identified  “quality  of  intelligence”  as  the  number  one  most  frequently  mentioned  analytic  concern, 
with  54%  of  respondents  identifying  the  issue  in  interviews.32  Mr.  Jeffrey  Cooper,  a  senior  Central 
Intelligence  Agency  (CIA)  analyst  has  identified  the  analytical  pathologies  that  permeate  all  levels 
of  analysis  across  the  entirety  of  the  IC.33  He  and  many  others  have  emphasized  and  praised  the 
standards  found  in  the  professional  practices  of  law,  medicine,  and  science  and  offer  tools,  both 
practical  and  cognitive,  which  can  be  leveraged  to  improve  the  tradecraft  of  analysis.  David 
Moore,  career  senior  analysts  at  the  National  Security  Agency  (NS A),  authored  both  2007  Critical 
Thinking  and  Intelligence  Analysis  and  2011  Sensemaking:  A  Structure  For  An  Intelligence 
Revolution,  arguing  that  critical  thinking  needs  to  be  woven  into  the  process  of  analysis  in  order  to 
maximize  an  analyst’s  ability  to  produce  effective  intelligence  assessments  while  minimizing  the 
potential  for  intelligence  failures.  Another  prolific  author  in  the  field  is  Richard  Heuer.  Heuer,  a 
CIA  veteran  whose  research  on  structure  analytical  techniques  (SAT)  is  the  foundation  for  the  Air 
Force’s  14N  Intelligence  Officer  analysis  teachings,  emphasizes  separate  analytical  techniques  that 
range  across  eight  different  categories  of  analysis,  thus  informing  much  of  the  research  to  follow. 
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Doctors  Daniel  Zelik,  Emily  Patterson,  and  David  Woods  from  Ohio  State  University 
published  two  Department  of  Defense  sponsored  papers,  Understanding  Rigor  in  Information 
Systems  and  Judging  Sufficiency:  How  Professional  Intelligence  Analysts  Assess  Analytical  Rigor, 
which  highlight  the  importance  of  analytical  rigor  and  evaluate  how  sufficiency  of  rigor  is 
perceived  and  communicated.34  These  papers  explore  the  attributes  of  analytical  rigor,  and 
describe  rigor  as  building  on  a  process-driven  approach  of  standardized  adherence  and  achieved 
sufficiency.  The  researchers  built  an  attribute-based  metric  to  assess  analytical  rigor  against  eight 
unique  attributes  as  shown  in  Figure  1  below. 


Figure  1.  Visualization  of  Zelik,  et  al.  Rigor  Metric  (Adapted  from  Understanding  Rigor  in  Information 
Analysis,  2007) 


Zelik  et  al.  underscore  the  pressures  driving  analysts  to  low  rigor  analysis,  including  data 
rich  overload  and  production  pressure.  In  turn,  it  is  extremely  difficult  for  decision  makers  to 
recognize  insufficient  rigor  balanced  against  a  given  decision  and  the  consequences  of  failure.  The 
above  rigor  metric  helps  frame  an  understanding  of  analytical  rigor,  which  can  be  leveraged  to 
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reduce  shallow  analysis  by  bolstering  and  conveying  analytical  rigor  visually.  Zelik  et  al. 
recommend  a  “participatory  exchange  model”  in  which  the  analysts  and  decision  makers  have  a 
dialogue  on  analytical  rigor  as  opposed  to  a  one-way  transfer  of  information.  Zelik  et  al.  apply  this 
model  to  a  briefing  type  exchange,  but  a  logical  inference  would  suggest  that  conveying  analytical 
rigor  along  with  written  intelligence  reports  would  also  have  similar  value. 

Advanced  analysis  brings  a  high  level  of  analytical  rigor  and  applied  tradecraft  in  order  to 
fulfill  sufficiency  to  the  above  stated  measurables.  To  achieve  rigor,  both  critical  thinking  and 
structured  analytic  techniques  (SATs)  should  be  applied.  Structured  analytic  techniques  are  used 
to  challenge  assessments,  identify  cognitive  bias,  stimulate  creativity,  and  measure  uncertainty.35 
SATs  include  mental  network  analysis,  structured  brainstorming,  analysis  of  competing  hypothesis, 
red  hat  analysis,  pre-mortem  analysis,  devil’s  advocacy,  and  structured  debate  to  name  a  few.36 
These  techniques  fall  into  broad  categories  like  visualization,  decomposition,  challenge  analysis, 
idea  generation,  hypothesis  generation/testing,  and  others.  SATs  are  a  process  driven  approach  that 
when  combined  with  intuition,  critical  thinking,  and  subject  matter  expertise  reduce  analytical  error 
and  promote  quality,  effective,  and  rigorous  analysis. 

Section  3:  SUFFICIENCY  OF  ANALYTICAL  RIGOR 

ICD  203  and  Joint  Publication  2-0,  Joint  Intelligence  both  articulate  an  expectation  of 
analytical  rigor  required  to  facilitate  the  intelligence  process,  but  in  close  review  of  these 
documents,  the  expected  level  of  rigor  is  not  specifically  defined,  quantified,  or  communicated.37 
They  have  not  provided  any  process  or  training  required  to  achieve  this  standard,  nor  have  they 
provided  the  mechanism  for  analysts  to  communicate  a  measure  of  applied  rigor.  This  is  not  to 
say  that  standards  do  not  exist;  in  ICD  203  tradecraft  standards  are  classified  into  five  “Analytic 
Standards”  which  include  nine  “Analytic  Tradecraft  Standards.”38  ICD  203  introduces  these  broad 


11 


categories  of  expectations  in  just  a  two-page  overview  and  goes  no  further.39  A  literary  review 
quickly  provides  a  depth  of  academic  thought  on  how  things  like  structured  analytical  techniques, 
estimative  language,  sourcing,  assumption  checks,  critical  thinking,  alternative  hypotheses 
exploration,  should  be  applied,  but  the  academic  explanation  does  not  translate  well  to  regulatory 
guidance.  Derived  from  and  similar  to  ICD  203,  AFI  14-133  again  labels  and  categorizes  analytic 
standards,  but  does  little  else  to  elaborate  on  the  processes  required  to  facilitate  and  achieve 
success.  Additionally,  AFI  14-133  uses  slightly  different  terminology  than  ICD  203,  showing  that 
the  Air  Force’s  attempt  to  convey  and  uphold  standards  breaks  down  quickly  as  the  two  documents 
are  unable  to  agree  on  which  standards  to  uphold  and  how  they  should  be  labeled.  For  example, 
the  Air  Force  has  injected  confusion  by  introducing  “Tenants”  in  the  place  of  ICD  203’s  “Analytic 
Standards”,  and  using  “Integrity”  in  place  of  “Independent  of  political  consideration.”  To  highlight 
the  differences  between  ICD  203  and  AFI  14-133,  Table  1  below  lists  the  standards  as  conveyed  in 
the  respective  documents.  This  example  shows  how  quickly  ICD  203 ’s  standards  migrated  with 
the  publication  of  the  Air  Force  Instruction.  How  the  remaining  15  members  of  the  IC  implement 
ICD  203 ’s  standards  were  not  analyzed. 
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ICD  203  Analytic  Standards 

Analytic  Standards 

a.  Objective 

b.  Independent  of  political  consideration 

c.  Timely 

d.  Based  on  all  available  sources  of  intelligence 
information 

e.  Implements  and  exhibits  Analytic  Tradecraft 
Standards,  specifically: 

(1)  Properly  describes  quality  and  credibility  of 
underlying  sources,  data,  and  methodologies 

(2)  Properly  express  and  explains  uncertainties 
associated  with  major  analytic  judgments 

(3)  Properly  distinguishes  between  underlying 
intelligence  information  and  analysts’ 
assumptions  and  judgments 

(4)  Incorporates  analysis  of  alternatives 

(5)  Demonstrates  consumer  relevance  and  address 
implications 

(6)  Uses  clear  and  logical  argumentation 

(7)  Explains  change  to  or  consistency  of  analytic 
judgments 

(8)  Makes  accurate  judgments  and  assessments 

(9)  Incorporates  effective  visual  information 


AFI 14-133  Intelligence  Analysis 

AF  Intelligence  Analysis  Tenets 

3.1.1.  Objectivity 

3.1.2.  Integrity 

3.1.3.  Synthesis 

3.1.4.  Collaboration 

3.1.5.  Anticipation 

3.1.6.  Requirements 

AF  Intelligence  Analysis  Standards 

3.2.1.  Timeliness 

3.2.2.  Appropriate  Sourcing 

3.2.3.  Accuracy 

3.2.4.  Level  of  Confidence 

3.2.5.  Assumptions  vs.  Judgements 

3.2.6.  Analysis  of  Alternatives 

3.2.7.  Relevance 

3.2.8.  Logical  Argumentation 

3.2.9.  Utility 

3.2.10.  Customer  Engagement 


Table  1.  ICD  203  Standards  compared  to  AFI  140-133  Standards 

Additionally,  the  AFI  14-202  Intelligence  series  provides  regulatory  guidance  to  the  Air 
Force  intelligence  enterprise  across  three  independent  volumes;  Training,  Standards  and 
Evaluation,  and  General  Intelligence  Rules,  but  does  not  expand  upon  or  reference  the  overarching 
intelligence  standards  that  have  been  directed  by  the  Office  of  the  Director  of  National  Intelligence; 
ICD  203.  The  ODNI  attempts  to  provide  the  IC  with  the  principles  of  analysis  through  a  top  down 
application  of  analytical  standards,  but  these  standards  do  not  penetrate  to  line  unit  analysts.  With 
AFI  14-133  introducing  the  service  to  analytic  standards,  the  regulation  falls  well  short  of  defining 
and  delineating  analytical  rigor.  This  regulation  suggests  to  its  readers  that  one  must  meet  all  of 
the  standards,  but  fails  to  provide  measures  of  success.  The  ODNI’s  Rating  Scale  for  Evaluating 
Analytic  Tradecraft  Standards  provides  what  the  Air  Force  has  failed  to  adapt,  it  acknowledges  that 
analysis  is  not  binary;  that  there  are  variations  in  quality  and  sufficiency.  The  ODNI  rating  scale  is 
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intended  primarily  for  ODNI  evaluators  to  conduct  post-production  evaluations.  This  document 
provides  amplified  guidance  for  evaluators  and  assists  them  in  judging  ICD  203  compliance  on  a 
scale  of:  Poor  (0),  Fair  (1),  Good  (2),  and  Excellent  (4).  ODNI  evaluators  provide  post-production 
quality  control  for  review  purposes.  This  information  is  currently  not  shared  with  intelligence 
consumers  or  decision  makers,  thus  having  a  tool  to  provide  consumers  a  rating  of  effective  rigor 
and  analytic  compliance,  but  not  sharing  the  information.  The  historical  failures  previously 
discussed  in  this  paper  have  shown  that  when  decision  makers  perceive  a  sufficient  amount  rigor, 
yet  the  rigor  is  weak,  it  can  result  in  catastrophic  consequences. 

Expected  dialog  between  analysts  and  consumers  is  implied  in  the  evaluation  and  feedback 
section  of  the  intelligence  process,  but  this  feedback  process  has  limited  structure.  At  best,  an 
intelligence  product  goes  out  and  feedback  comes  back.  There  are  many  mechanisms  to  facilitate 
this  type  of  feedback,  but  this  does  not  meet  the  expectation  of  dialogue.  Dialogue,  as  opposed  to 
monologue,  requires  two-way  communication.  There  is  an  expectation  by  the  intelligence 
consumer  that  the  analysis  being  presented  is  sufficiently  rigorous  yet  there  is  currently  no 
systematic  mechanism  or  standard  for  an  analyst  to  quantify,  convey,  and  certify  analytical  rigor 
prior  to  publication  of  intelligence  products.  The  IC’s  current  evaluation  capabilities  are  in 
compliance  with  the  Intelligence  Reform  and  Terrorism  Prevention  Act  of 2004  requirements  to 
conduct  postproduction,  after  action  reviews.  Rigor  and  accuracy  are  mutually  exclusive;  one  does 
not  guarantee  the  other.  This  is  understood  in  the  tradecraft,  and  indeed  the  ICD  203  analytic 
tradecraft  standard  requiring  analysts  to  “properly  express  and  explain  uncertainties  associated  with 
major  analytic  judgments”  was  established  to  standardize  how  confidence  levels  and  expressions  of 
likelihood  are  conveyed  to  consumers,  which  is  different  than  conveying  rigor.  It  is  safe  to  say  that 
national  policymakers  consuming  a  National  Intelligence  Estimate  would  expect  high  rigor,  where 
a  combatant  commander  digesting  current  intelligence  might  accept  the  risk  of  low  analytical  rigor 
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given  time  constraints  or  limited  corroboration.  High  rigor  analysis  can  appropriately  result  in  a 
low  confidence  assessment. 


Section  4:  PROPOSAL 

This  paper  will  analyze  the  impacts  and  costs  of  establishing  an  ODNI  center  of  excellence, 
a  schoolhouse  designed  to  train  and  certify  Masters  of  Analytic  Tradecraft  (MAT)  analysts.  The 
primary  purpose  of  MAT  analysts  is  to  evaluate  and  rate  intelligence  products  for  adherence  to  ICD 
203  Analytic  Standards  and  to  assess  and  convey  analytical  rigor  to  production  analysts  and 
intelligence  consumers.  The  ODNI’s  Analytic  Integrity  and  Standards  group  (AIS)  currently  has 
evaluation  and  rating  processes  that  are  employed  to  assess  post  publication  intelligence  as  spot- 
check,  quality  control,  and  review  mechanisms  in  accordance  with  section  1019  of  the  Intelligence 
Reform  and  Terrorism  Prevention  Act  of 2004,  “shall  perform,  on  a  regular  basis,  detailed  reviews 
of  finished  intelligence  product(s).”40  MAT  analysts  and  their  evaluation  capability  would  differ 
from  and  potentially  complement  existing  AIS  processes  in  that  MAT  analysts  would  1)  be 
embedded  at  the  unit  and  organizational  levels,  2)  evaluate  intelligence  reporting  prior  to 
publication,  3)  exponentially  robust  current  efforts  in  manpower  and  capability,  and  4)  culturally 
build  a  cadre  across  the  Intelligence  Community  that  is  focused  of  ICD  203  compliance  and  the 
adherence  to  analytical  rigor  expectations.  Certified  MAT  analysts  will  be  effectively  trained  to 
certify  the  level  of  analytical  rigor  prior  to  publication  of  intelligence  publications.  This 
certification  will  accompany  the  intelligence  report  in  the  form  of  a  unique  and  branded  label,  thus 
providing  a  simple,  standardized,  and  easily  recognizable  conveyance  of  analytical  rigor  that  is 
stamped  with  the  license  number  of  the  MAT  analysts  thus  communicating  an  authoritative 
assessment  of  analytical  rigor  and  adherence  to  ICD  203  Analytic  Standards. 
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Figure  2.  Proposed  MAT  Rating  Badges  (as  conceptualized  by  the  author) 

Evaluations  and  MAT  certifications  of  analytical  rigor  will  be  a  unique  capability  of 
certified  MAT  analysts.  Only  certified  MAT  analysts  will  be  empowered  to  evaluate  and  certify 
intelligence  products.  The  following  analysis  will  determine  if  an  AIS  sponsored  schoolhouse, 
certified  MAT  analysts,  certified  intelligence  products  would  or  would  not  increase  adherence  to 
ICD  203  standards  and  improve  the  culture  of  analytical  rigor  across  IC. 

Section  5:  RESEARCH  METHODOLOGY 

The  following  analysis  evaluates  the  above  proposal  utilizing  the  following  criteria: 

Evaluation  Criterion  1:  Effectiveness 

Criterion  one  will  analyze  how  effective  given  proposals  might  be  towards  achieving  the 
goals  of  promoting  and  conveying  analytical  rigor  across  the  IC,  ensuring  that  the  rigor  is 
standardized,  and  assessing  how  effective  a  given  proposal  might  be  in  facilitating  communication. 
It  is  recognized  that  given  the  classified  nature  of  the  materials  and  the  limited  amount  of 
unclassified  data,  effectiveness  will  be  difficult  to  measure. 

Evaluation  Criterion  2:  Implementation 

The  proposal  will  be  evaluated  against  identified  barriers  to  implementation.  Along  this 
line,  the  evaluation  will  strive  to  understand  and  identify  barriers  such  as  training,  manpower,  and 
support  requirements.  From  a  resourcing  perspective,  proposals  will  be  evaluated  for  cost 


16 


effectiveness  and  will  explore  resourcing  implications. 

Evaluation  Criterion  3:  Adaptation 

The  adaption  is  the  final  criterion  by  which  proposed  processes  will  be  evaluated.  Factors 
effecting  adaption  include  consensus,  leadership  buy-in,  cooperative  elements,  feasibility,  target 
population,  performance  measures,  and  goals.  Given  the  size  of  the  IC,  the  paper  will  evaluate 
how  long  it  will  take  to  implement  the  given  proposals  and  subsequently  evaluate  how  long  it  will 
take  to  effectively  penetrate  the  proposals  to  the  unit  level. 

Section  6:  ANALYSIS  /  EVALUATION 

Evaluation  Criterion  1:  Effectiveness 

Evaluating  the  certification  proposal  against  existing  processes  suggests  this  would  be  an 
effective  approach  on  multiple  levels.  By  establishing  a  center  of  excellence  facility  to  train  MAT 
analysts  under  the  authority  of  the  ODNI  would  be  another  step  in  consolidating  a  fragmented 
analytic  community  that  was  so  aptly  described  in  the  9-11  Commission  Report:  “It  is  hard  to 
“break  down  stovepipes”  when  there  are  so  many  stoves  that  are  legally  and  politically  entitled  to 
have  cast-iron  pipes  of  their  own.”41  The  ODNI  currently  sponsors  two  training  initiatives  that  can 
be  used  as  a  model  for  MAT  analysts  training.  The  ODNI’s  Analysis  101/201  courses  are  intended 
for  relatively  new  analysts  and  focus  on  the  fundamentals  of  analysis  and  analytical  collaboration 
respectively.  By  building  intelligence  product  evaluators  who  are  trained  and  certified  to  evaluate 
on  ICD  203  standards,  MAT  training  would  further  promote  a  Joint  Intelligence  Community 
culture  verses  a  culture  of  independent  CIA,  NSA,  DIA,  Air  Force  analysts.  Cultural  affiliation 
with  independent  intelligence  agencies  is  not  a  bad  thing  as  long  as  there  is  a  touchstone  to  a  larger, 
interconnected  culture  consolidated  under  the  ODNI  and  its  ICD  203  standards.  This  is  inline  with 
the  9-11  and  WMD  commission  recommendations  and  meets  the  underlying  intent  of  the  IRTPA 
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when  it  established  a  Director  of  National  Intelligence  and  the  ODNI’s  responsibilities. 

Having  ODNI  certified  MAT  analysts  to  evaluate  and  certify  intelligence  products  prior  to 
production  would  be  an  unprecedented  approach  at  the  scale  that  is  being  suggested.  Current 
quality  assurance  efforts  are  fragmented  and  scatted  across  the  entirety  of  the  IC.  As  AFI  14-133 
has  demonstrated,  ICD  203  interpretation  evolves  as  the  organizational  process  gets  further  away 
from  the  originating  source.  To  certify  MAT  analysts  as  evaluators  would  consolidate  the  tribe  and 
extend  the  ODNI’s  ICD  203  penetration  into  the  disparate  organizations  and  agencies  of  the  IC. 
Given  that  many  organizations  already  have  standards  and  evaluation  (StanEval)  or  quality  control 
(QC)  functions  established,  a  certification  process  would  take  the  same  billets  and  efforts  and  align 
them  across  the  IC.  If  an  IC  component  wanted  to  exceed  a  given  standard,  there  would  be  no 
resistance,  but  a  foundational,  tiered  evaluation  and  analytical  rating  scale  would  be  the  baseline. 
Unifying  the  IC  efforts  under  one  recognized  certification  process  would  be  healthy  as  it  relates  to 
establishing  a  feedback  mechanism  and  evaluating  and  promoting  analytical  rigor,  thus  benefiting 
intelligence  consumers  and  analysts  alike.  Having  MAT  evaluators  certify  intelligence  products 
and  share  their  findings  would  be  revolutionary;  for  the  first  time  giving  decision  makers  an 
understanding  of  the  quality  of  analytical  rigor  that  informed  the  consumed  intelligence  products 
that  they  base  their  decision  making  processes  on.  This  would  effectively  empower  intelligence 
consumers  to  judge  sufficiency  for  themselves. 

Given  the  known  analysis  issues  leading  up  the  invasion  of  Iraq  and  the  level  of  dependence 
of  the  nation’s  decision  makers  on  the  2002  NIE,  it  is  assessed  that  had  the  NIE  been  accompanied 
by  a  pre-production  evaluation  of  analytical  rigor  and  adherence  utilizing  today’s  tradecraft 
standards,  then  decision  makers  would  have  been  informed  on  the  insufficiency  of  process  and  thus 
demanded  addition  rigor.  Instead,  decision  makers  perceived  rigor  and  falsely  trusted  the  analysis, 
resulting  an  over  two  trillion  dollar  expense  and  almost  190,000  lives  lost  over  the  course  of  the 
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Iraq  conflict.42  This  is  not  to  suggest  that  a  MAT  Analysts  certification  alone  would  have 
prevented  the  war  in  Iraq.  There  were  many  issues  identified,  many  of  which  were  addressed  in  the 
2004  IRTPA,  but  a  MAT  certification  of  the  2002  NIE  would  have  been  a  valuable  tool  in 
communicating  sufficiency  of  analytical  rigor. 

MAT  evaluators  would  be  vulnerable  to  error,  thus  requiring  continued  training.  To  carry 
authority,  MAT  analysts  would  need  to  come  to  this  training  seasoned  by  years  of  analytical 
experience.  MAT  evaluators  would  require  currency  training,  and  a  mechanism  to  communicate 
and  collaborate  with  senior  evaluators.  Evaluations  would  need  to  be  reviewed  for  quality  control 
and  adherence  to  yet  to  be  established  or  vetted  processes. 

The  credentials  of  MAT  evaluators  would  continually  be  tested.  Mistakes  could  damage 
the  MAT  “brand”  and  in  turn  devalue  the  initiative.  Trust  in  the  MAT  brand  would  require  time,  a 
consistent  product,  and  diligent  individuals  all  of  which  could  be  variable  and  undermined  by 
missteps  as  the  process  is  created. 

Evaluation  Criterion  2:  Implementation 

Implementation  of  a  MAT  analyst  training  and  certification  process  will  have  challenges. 
For  this  analysis  I  assumed:  1)  The  course  will  be  one  month  long,  tripling  the  current  Analysis  101 
timeframe  due  to  the  expanded  syllabus  and  certification  requirements.  2)  The  class  size  will  be 
twenty-five  which  is  a  typical  classroom  size  for  this  type  of  instructor  lead  engagement  and  is  also 
modeled  after  the  approximate  ODNI  Analysis  101  class  size.43 

The  barriers  to  implementation  are  training,  manpower,  and  support  requirements.  For  a 
schoolhouse  to  be  established,  facilities,  instructor  cadre,  administrative  overhead,  syllabus 
development  would  all  need  to  be  established.  To  stand  up  a  schoolhouse  would  require  a 
significant  amount  of  funding.  Fortunately,  much  of  this  work  has  already  been  accomplished. 
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The  ODNI’s  AIS  already  has  a  means  for  training  evaluators  and  amplified  guidance  foe  evaluators 
already  exists  in  the  form  of  the  ODNI’s  Rating  Scale  for  Evaluation  Analytic  Tradecraft 
Standards.  Instructor  cadre  and  infrastructure  are  already  established  at  the  Chantilly,  VA  training 
facility.  The  current  efforts  would  need  to  be  scaled  at  a  cost  that  would  require  further  evaluation 
beyond  to  scope  of  this  paper.  If  the  return  on  investment  of  the  Analysis  101  course  is  deemed 
cost  effective,  a  MAT  certification  would  be  a  double  down  on  this  same  investment.  Most 
organization  across  the  IC  offer  some  form  of  analytical  training,  the  Air  Force  offers  such  a  class 
at  Goodfellow  Air  Force  Base.  If  the  Air  Force  were  to  dissolve  its  service  specific  analysis 
training  in  trade  for  Joint,  ODNI  sponsored  training  then  the  cost  of  implementation  would  be 
negligible  from  the  Air  Force’s  perspective. 

A  2008  RAND  report  on  Assessing  the  Tradecraft  of  Intelligence  Analysis  recommended 
instituting  IC-wide  tradecraft  training,  and  was  quick  to  acknowledge  both  the  need  for  and  the 
difficulty  of  implementing  such  Community-wide  training.  As  compared  to  the  RAND 
recommendations,  this  proposal  is  surgical  in  its  execution.  One  given  benchmark  of  success  is  the 
ODNI  Analysis  101  course,  which  has  graduated  over  7,000  students  from  over  thirty  different 
intelligence  and  law  enforcement  organizations  since  its  conception  in  February  of  2007. 44  A 
similar  scale  of  effort  would  be  appropriate  for  a  MAT  certification  program,  with  the  goal  of 
providing  MAT  trained  analysts  embedded  at  various  organizational  levels. 

Manpower  being  a  valued  commodity,  it  is  not  expected  that  additional  billets  funded  on 
unit  manning  documents  for  MAT  analysts.  Therefore  this  capability  and  function  would  have  to 
be  absorbed  by  an  already  stressed  of  manpower.  Most  organizations  have  a  section  dedicated  to 
StanEval  or  QC,  but  currently  these  functions  are  not  aligned  to  the  ODNI  and  ICD  203  analytic 
standards.  For  implementation  to  be  successful,  IC  elements  would  have  to  shift  current  StanEval 
and  QC  functions  to  align  with  ODNI  direction.  In  many  ways  this  would  require  disparate 
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elements  across  the  IC  to  defer  some  control  to  the  ODNI,  which  would  be  politically  challenging. 
For  example,  AFI  14-133  defines  the  Air  Force’s  established  analytic  standards,  which  differ  from 
established  ODNI  standards.  For  this  proposal  to  be  successful,  Air  Forces  standards  would  have 
to  defer  to  ODNI  standards.  The  Air  Force  could  implement  higher  standards  if  it  chooses,  but 
could  not  relax  those  standards  without  the  consent  of  the  ODNI. 

Instead  of  forcing  MAT  certified  analysts  upon  the  entirety  of  the  IC,  one  implementation 
approach  would  be  to  offer  MAT  certification  as  an  enhancement  to  current  unit  level  processes. 

IC  organizations  that  elect  to  invest  in  and  train  MAT  certified  analysts  would  then  be  allowed  to 
self-evaluate  and  certify  their  intelligence  products  with  the  branded  MAT  certification  badges 
offered  above.  Those  organizations  electing  to  opt  out  would  not  be  able  to  certify  their  products. 

Evaluation  Criterion  3:  Adaptation 

Given  the  exclusive  and  unique  qualities  of  the  MAT  analyst’s  ability  to  evaluate  and  label 
intelligence  with  branded  rating  labels,  it  is  possible  that  once  intelligence  consumers  become 
aware  of  MAT  certification,  they  would  be  a  driving  force  in  encouraging  implementation,  but  this 
is  a  currently  an  unknown  demand.  It  is  foreseeable  that  reporting  intelligence  agencies  would 
want  this  capability  as  a  way  to  laud  their  own  due  diligence  and  adherence  to  ODNI  established 
processes.  It  is  also  foreseeable  that  intelligence  consumers,  once  educated  on  the  MAT 
certification  program,  will  push  for  the  intelligence  products  that  they  consume  to  be  pre-evaluated 
and  MAT  rated  for  rigor  and  standard  adherence.  Given  that  MAT  certification  is  additive  in 
nature,  resistance  from  intelligence  consumers  is  not  likely,  but  since  the  burden  of  certifying 
products  falls  to  the  intelligence  producers  it  is  foreseeable  that  analysts  and  organizations  would 
push  back  if  they  are  unable  to  see  the  value.  If  MAT  certification  were  to  become  coveted  by 
senior  analysts  and  organizational  leaders,  then  buy-in  from  both  analysts  and  decision  makers 
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would  be  likely. 

A  phase-in  approach  would  provide  minimal  risk  on  investment,  with  scale  and  IC 
penetration  growing  as  MAT  analysts  obtain  certification  and  initiate  the  evaluation  of  their 
organization’s  intelligence  assessments.  It  is  foreseeable  that  once  products  are  accompanied  by 
analytical  rigor  ratings,  intelligence  consumers  will  request  the  highest  ratings  possible  of  the 
products  they  are  digesting.  When  MAT  ratings  fall  below  the  intelligence  consumer’s  sufficient 
requirements  for  analytic  rigor  this  could  facilitate  further  and  potentially  healthy  dialogue  between 
IC  components  and  consumers  concerning  such  topics  as  resources,  capabilities,  time  requirements, 
collection,  analysis  and  depth  of  expertise. 

Adapting  MAT  certified  analysts  would  not  require  day  one,  mass,  IC-wide  adaptation.  A 
measured  and  steady  implementation  of  the  proposal  would  be  a  feasible.  No  changes  to  the 
current  intelligence  process  would  be  required  with  MAT  certification  being  an  additive 
supplement  to  established  baseline  intelligence  production.  Minimal  penetration  could  still  present 
positive  results.  Success  would  beget  success.  If  MAT  analysts  conducting  per-publication 
evaluations  proves  to  be  a  failure,  then  the  investment  lost  would  be  measured  and  relative  to  the 
number  of  graduates.  The  first  year  of  implementation  could  be  considered  a  beta,  proof  of 
concept,  thus  further  managing  expectations  as  some  of  the  trial  and  error  implementation  is 
worked  out. 

Where  establishment  of  MAT  analysts  would  robust  existing  processes,  the  cultural  change 
and  adaptation  across  17  unique  IC  organizations  and  agencies  would  be  difficult.  ODNI  directives 
would  need  to  be  written  and  enforced  to  ensue  IC-wide  adaptation.  It  has  taken  multiple 
congressional  investigations  and  the  passing  of  the  2004  IRTPA  to  get  to  the  two  and  a  half  pages 
of  analytic  standards  outlined  in  ICD  203,  to  see  MAT  certification  fully  adapted  would  require 
more  that  IC-wide  buy-in,  it  would  require  mandated  adherence. 
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Section  7:  RECOMMENDATIONS 


Modeling  after  the  civilian  sector’s  U.S.  Green  Building  Council’s  (USGBC)  Leadership  in 
Energy  and  Environmental  Design  (LEED)  program  and  borrowing  from  the  medical  and  legal 
professions,  it  is  recommended  that  the  ODNI  establish  a  certification  board  process  to  credential 
analysts  as  Masters  of  Analytic  Tradecraft  (MAT).  MAT  analysts  would  be  certified  to  employ  the 
ODNI’s  analytic  rating  scale,  evaluate  analytic  products,  and  certify  the  rigor  of  those  products 
prior  to  publication.  To  become  MAT  certified,  analysts  would  need  to  attend  a  centralized 
training  schoolhouse.  This  center  of  excellence  would  be  sponsored  by  the  ODNI,  thus  being  joint 
in  nature  and  linked  across  the  IC.  Having  this  level  of  consolidated  analytical  tradecraft  training 
at  the  top  of  the  IC,  with  a  joint  cadre  of  senior  analysts  teaching  curriculum  would  be  a  top  down 
approach  currently  lacking  in  the  IC.  Graduation  would  require  the  passing  of  a  board  style 
examination.  Although  ICD  203  has  provided  unifying  direction,  it  is  this  author  of  this  paper’s 
opinion  that  resultant  training  opportunities  continue  to  be  disjointed,  stove-piped,  and  fragmented 
across  agency  lines,  resulting  in  poorly  disseminated  standards  and  divergent  lines  of  effort.  An 
IC,  enterprise-wide  analytical  center  of  excellence  is  needed  to  raise  the  focus  on  analysis  and 
analytical  tradecraft.  This  schoolhouse  would  be  the  driving  force  behind  standards,  collaboration, 
continued  learning,  the  sharing  of  best  practices,  quantifying  and  communicating  analytical  rigor, 
and  certifying  MAT  analysts. 

The  curriculum  of  MAT  certifying  course  would  be  focused  on  both  the  application  and  the 
evaluation  of  analytic  tradecraft  standards.  This  would  be  different  from  existing  agency  and 
service  specific  analysis  training  courses  in  that  MAT  analysts  would  graduate  uniquely  certified  as 
Office  of  the  Director  of  National  Intelligence  analytic  tradecraft  evaluators  and  thus  able  to 
evaluate  assessments  and  rate  associated  analytical  rigor.  This  would  take  what  is  currently  a  post 
report  quality  control  spot  check  that  is  conducted  by  a  small  portion  of  ODNI  staff  and  turn  it  into 


23 


an  IC,  enterprise- wide  self-evaluation  of  analytical  rigor  that  happens  prior  to  an  intelligence  report 
ever  being  published. 

By  establishing  a  certification  process,  a  MAT  analyst  would  be  trained  and  empowered  (by 
virtue  of  their  unique  ODNI  certification)  to  uphold  established  standards.  Quantifying  and 
evaluating  the  level  of  analytical  rigor  that  informed  an  organization’s  assessment  and  publishing 
the  resultant  rating  in  the  intelligence  report  will  inform  decision  makers  the  amount  and  quality  of 
analytical  rigor  that  informed  the  analysis  upon  which  they  are  trusting.  It  is  then  intelligence 
consumer’s  responsibility  to  determine  if  the  rated  level  of  rigor  is  sufficient  for  their  needs.  Due 
to  the  overt  evaluation  of  intelligence  products,  this  rating  system  would  award  analysts  to  strive 
for  high  levels  of  rigor. 

It  is  recommended  that  the  IC  mimic  the  LEED  model,  certifying  MAT  analysts  to  evaluate 
their  assessments  for  analytical  rigor  and  standardization  by  applying  their  training  and  utilizing  the 
ODNI’s  Rating  Scale  for  Evaluating  Analytic  Tradecraft  Standards.  The  resultant  evaluation 
findings  would  accompany  the  assessment.  This  rating  could  only  be  given  by  certified  MAT 
analysts  and  would  be  branded  in  a  standardized  way  as  to  convey  the  amount  of  analytical  rigor 
that  went  into  the  published  intelligence  report.  Similar  to  how  USGBC  uses  LEED  certifications 
to  convey  a  level  of  environmental  performance  and  adherence  to  standards  and  established 
practices,  so  would  MAT  certification  convey  a  level  of  analytical  rigor  to  the  intelligence 
consumer.  In  the  case  of  analysis,  MAT  analysts  would  measure  and  certify  rigor  using  the  ODNI 
rating  scale,  and  convey  the  findings  utilizing  a  conformed  branding  scale.  Departing  from  the 
current  ODNI  scale  of  poor,  fair,  good,  and  excellent,  the  badges  displayed  above  (Figure  1) 
envision  a  level  one  through  four  criteria  with  level  one  equating  to  low  rigor  and  building  up  to  a 
level  four  which  would  equate  to  excellent  rigor.  Similar  to  how  a  licensed  engineer  might  stamp 
their  work  with  a  seal  containing  their  license  number,  so  it  is  recommended  that  MAT  analysts 
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certify  their  evaluations  by  including  their  assigned  MAT  number. 

Currently  ODNI  encourages  analysts  across  the  IC  to  become  familiar  with  the  ODNI  rating 
scale  as  a  way  to  enhance  analytic  tradecraft.  The  MAT  certification  proposal  goes  much  further, 
intensively  training  MAT  analysts  to  be  evaluators  of  ICD  203  analytic  tradecraft  standards,  having 
intelligence  reports  assessed  for  analytical  rigor  before  publication,  communicating  that  rigor  to 
intelligence  consumers,  and  providing  an  immediate  feedback  loop  to  the  analysts  who  are 
presenting  their  analysis.  By  embedding  MAT  analysts  at  the  unit  level  and  connecting  them  to  the 
intelligence  production  process,  the  ICD  203  standards  would  be  effectively  pushed  down  and 
permeated  across  the  IC. 

An  additional  requirement  to  maintain  MAT  certification  would  be  to  complete  and  track 
continuing  education  (CE).  Keeping  with  the  LEED  model,  MAT  analyst  would  be  required  to 
earn  30  CE  hours  every  two  years.45  Qualifying  CE  opportunities  would  be  defined  and  tracked  by 
ODNI’s  Analytic  Integrity  and  Standards  group  and  thus  ensure  currency  and  proficiency  of  MAT 
analysts. 

Additional  benefits  of  establishing  MAT  certified  analysts  would  be  the  duel  hatted  nature 
of  these  analysts  as  both  evaluators  of  analytic  products  and  as  embedded  subject  matter  experts 
expected  to  act  as  analysis  tradecraft  instructors  and  leaders.  MAT  analysts  would  provide 
advanced  unit  level  instruction  on  ICD  203 ’s  nine  analytic  tradecraft  standards.  Beyond  their 
individual  instructor  roles,  a  community  of  close-knit  MAT  analysts  would  be  developed,  thus 
facilitating  effective  cross  talk  and  communication.  ODNI  would  use  MAT  analysts  to  maintain 
dialogue,  convey  best  practices,  provide  feedback  avenues,  disseminate  time  sensitive  bulletins, 
build  and  maintain  unit  level  libraries,  capture  and  convey  tactics,  techniques,  and  procedures  as 
they  pertain  to  analysis,  and  provide  analytical  experts  who  conduct  post-product  after  action 
reviews  and  assess  where  analysis  went  right  and  where  it  came  up  short.  MAT  analysis  will  be 
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unit  and  organizational  level  focal  points,  ensuring  ICD  203  integration  and  coordination.  MAT 
analysts  could  also  fill  an  advisory  role  to  intelligence  production  commanders  and  leaders, 
ensuring  that  the  need  for  analytical  rigor  has  a  voice  amongst  the  myriad  of  competing  force: 
manning,  resources,  and  production  rates,  to  name  a  few.  Additionally,  MAT  analysts  could  liaise 
between  intelligence  consumers  and  production  analysts  to  ensure  feedback  is  provided  and 
absorbed. 

In  addition  to  the  civil  sector  professions  that  require  certification  (doctors,  lawyers, 
engineers,  LEED  associates,  etc.),  the  Air  Force’s  Weapons  Instructor  Courses  (WIC)  provides 
another  model,  which  could  be  adapted  by  the  ODNI  and  applied  to  MAT  certification.  The  Air 
Force’s  WIC  prides  itself  on  its  ability  to  develop  an  institutional  pool  of  tactical  and  operational 
experts.46  Embodying  a  creed  of  "humble,  approachable  and  credible"  the  aggregate  of  WIC 
graduates  form  a  fraternity  of  trusted  experts  and  valued  problem-solvers.47  Upon  completing  an 
intensive,  graduate-level  curriculum,  Air  Force  WIC  graduates  integrate  at  the  unit  level  and 
provide  a  wide  spectrum  of  academic  and  advisory  support.  They  facilitate  the  collection  of 
tactical  level  knowledge  and  use  it  to  inform  and  author  tactical  doctrine.  This  collective  of  the  Air 
Force’s  best  weapon’s  officers  forms  a  self-aware,  continually  improving,  and  always 
communicating  trust  of  knowledge  that  both  informs  and  enables  best  practices  across  the  multi- 
domain  tactical  environments  in  which  the  Air  Force  fights.48 

In  this  same  vein,  but  transcending  any  one  service  or  agency,  MAT  analysts  would  be 
ODNI  trained  and  credentialed,  providing  unit  and  organizational  level  experts  in  analytical 
tradecraft  and  informing  and  upholding  ICD  203 ’s  standards  of  analysis.  Empowered  as  evaluators 
of  analytical  rigor,  MAT  analysts  would  rate  assessments  utilizing  the  afore-mentioned  branded 
rating  system,  to  ensure  and  convey  rigor  while  concurrently  fulfilling  the  roles  of  trainer  and 
trusted  advisor. 
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Notes 


(All  notes  appear  in  shortened  form.  For  full  details,  see  the  appropriate  entry  in  the  bibliography) 
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